1. Field of the Invention
The present invention relates to a method of picking up a moving image by a camera and carrying out automatic stabilization control of the image picked up.
2. Description of the Background Art
Apparatuses for picking up moving images by cameras and carrying out automatic stabilization control of the images picked up have been conventionally proposed. For example, an apparatus disclosed in JP-A-63-166370 records pixel data in the preceding frame and calculates a motion vector by correlation with the current frame, to carry out automatic stabilization control.
In such a method, however, the automatic stabilization control may end in failure depending on the image. An example is a case where the image hardly patterned or a passing object which rapidly cuts across in front of the camera.
The applicant of the present invention has developed an automatic stabilization control method so adapted as to cause a user to designate a range of frames which are to be subjected to automatic stabilization control excluding frames whose automatic stabilization control may end in failure and subject only the frames in the designated range to automatic stabilization control.
The automatic stabilization control method developed by the applicant of the present invention will be described. The automatic stabilization control method has not been known yet.
FIG. 1 shows the procedure for entire processing of the automatic stabilization control method developed by the applicant of the present invention.
In the step S101, processing for causing a user to designate a range of frames which are to be subjected to automatic stabilization control (a correction frame range) from data representing an image is performed. In the step S102, an automatic stabilization control coefficient for carrying out automatic stabilization control is calculated for each of the frames within the correction frame range. In the step S103, an image area (common area) where the image is common among all the frames within the correction frame range is extracted from the automatic stabilization control coefficient, for each of the frames within the correction frame range, calculated in the step S102.
In the step 104, the frames are read one by one in a memory from a moving image file. In the step S105, it is judged whether or not the frame (target frame) read in the memory is a frame which is to be subjected to automatic stabilization control.
In a case where the target frame read in the memory is a frame which is to be subjected to automatic stabilization control, the frame is subjected to automatic stabilization control processing in the step S106, after which the procedure proceeds to the step S107. In the step S107, the frame after the automatic stabilization control is outputted to an output file.
On the other hand, in a case where the target frame read in the memory is not a frame which is to be subjected to automatic stabilization control, the procedure proceeds to the step S107 without subjecting the frame to automatic stabilization control processing. In the step S107, the frame is outputted as it is to an output file.
In the foregoing step S106, the automatic stabilization control is carried out while enlarging the target frame on the basis of an enlargement ratio calculated from the respective sizes of the common area extracted in the step S103 and the original image and an automatic stabilization control coefficient corresponding to the target frame calculated in the step S102.
In the step 108, it is judged whether or not the target frame is the final frame out of the input frames. The procedure is terminated in a case where the target frame is the final frame, while proceeding to the step S109 in the other case. In the step 109, the frame number of the target frame is updated by one. Thereafter, the procedure is returned to the step 104. In the step S104, the succeeding frame is read, to perform the processing in the step S105 and the subsequent steps again.
FIG. 2 shows the detailed procedure for the processing in the step 102 shown in FIG. 1 (processing for calculating an automatic stabilization control coefficient).
In the step S201, a frame number i is initialized to one in order to process the first frame within the correction frame range, to read in the memory the first frame within the correction frame range. In the step 202, the frame number i is increased by one, to read in the memory the succeeding frame.
In the step S203, a geometric conversion coefficient is calculated between the frame (target frame) read in the step S202 and the preceding frame. The geometric conversion coefficient will be described later. In the step S204, geometric conversion coefficients between the adjacent frames from the first frame within the correction frame range to the target frame are accumulated, to calculate the geometric conversion coefficient between the first frame within the correction frame range and the target frame. The geometric conversion coefficient obtained in the step S204 is an automatic stabilization control coefficient for subjecting the target frame to automatic stabilization control.
In the step 205, it is judged whether or not the target frame is the final frame within the correction frame range. When the target frame is the final frame within the correction frame range, the processing is terminated. In the other case, the procedure is returned to the step S202.
FIGS. 3a and 3b are diagrams for explaining the principle upon which automatic stabilization control is carried out by geometric conversion.
Two frames which are continuous in time shown in FIG. 3a indicate that although the same scene is picked up, the position thereof is shifted by automatic stabilization control. In order to carry out automatic stabilization control so that a subject is not moved, this is feasible by making geometric conversion between the frame number i-1 and the frame number i. The geometric conversion in the simplest case is a two-dimensional parallel movement, which is represented by a two-dimensional parallel movement as indicated by an arrow in FIG. 3b. In the more complicated geometric conversion, affine transformation, projective plane transformation, and so on are used.
The geometric conversion can be calculated by extracting a motion vector between the images. As to the geometric conversion coefficients calculated between the adjacent frames, when a coordinate axis is selected such that the first frame within the correction frame range is used as the basis, the geometric conversion coefficient between each of the frames within the correction frame range and the first frame within the correction frame range can be calculated by successively multiplying the geometric conversion coefficients, starting with the first frame within the correction frame range.
FIG. 4 is a diagram for explaining the processing in the step S103 shown in FIG. 1, that is, the processing for extracting the image area (common area) where the image is common among all the frames within the correction frame range from the automatic stabilization control coefficient for each of the frames within the correction frame range.
FIG. 4 illustrates a case of three frames. In order to carry out automatic stabilization control by geometric conversion to indicate that there is no invalid area in a peripheral part of an image, an image area where the image is common among all the frames within the correction frame range, indicated by hatching in FIG. 4, must be extracted. Further, the area is smaller in size than the input frame. Therefore, the area is made equal in size to the input frame by subjecting the image to enlargement processing.
However, the common area is subjected to enlargement processing for the frame which is to be subjected to automatic stabilization control, while not being subjected to enlargement processing in the frame which is not to be subjected to automatic stabilization control. Therefore, an enlargement ratio differs between the frame which is designated as one in the correction frame range and the frame which is not designated as one in the correction frame range. When viewed as the entire series of images, an unnatural joint occurs between the frame which has been subjected to automatic stabilization control and the other frame.